Invalid Permutation Tests

Invalid Permutation Tests

Permutation tests are often presented in a rather casual manner, in both introductory and advanced statistics textbooks. The appeal of the cleverness of the procedure seems to replace the need for a rigorous argument that it produces valid hypothesis tests. The consequence of this educational failing has been a widespread belief in a “permutation principle”, which is supposed invariably to give...

Valid inferences from invalid tests?

I read with interest the remarks of Reuben J. Silver, which were published in the November 2001 issue of the American Psychologist on the occasion of his receiving the Award for Distinguished Contributions to Applied Psychology as a Professional Practice. Silver took the opportunity to delineate his approach to the practice of professional psychology. He described a perspective that combines an...

Mantel Permutation Tests

Permutation Tests 101

Given independent samples from P and Q, two-sample permutation tests allow one to construct exact level tests when the null hypothesis is P = Q. On the other hand, when comparing or testing particular parameters θ of P and Q, such as their means or medians, permutation tests need not be level α, or even approximately level α in large samples. Under very weak assumptions for comparing estimators...

Permutation Tests for Classification

We introduce and explore an approach to estimating statistical significance of classification accuracy, which is particularly useful in scientific applications of machine learning where high dimensionality of the data and the small number of training examples render most standard convergence bounds too loose to yield a meaningful guarantee of the generalization ability of the classifier. Instea...